For many years, standardized tests have been administered to examinees for various reasons such as for educational testing or for evaluating particular skills. For instance, academic skills tests, e.g., SATs, LSATs, GMATs, etc., are typically administered to a large number of students. Results of these tests are used by colleges, universities and other educational institutions as a factor in determining whether an examinee should be admitted to study at that particular institution. Other standardized testing is carried out to determine whether or not an individual has attained a specified level of knowledge, or mastery, of a given subject. Such testing is referred to as mastery testing, e.g., achievement tests offered to students in a variety of subjects and the results being used for college credit in such subjects.
FIG. 1 depicts a sample question and related directions which might be given on a standardized test. The stem 12, the stimulus 14, responses 16, and directions 18 for responding to the stem 12 are collectively referred to as an item. The stimulus 14 is the text and/or graphical information, e.g., a map, scale, graph, or reading passage, to which a stem 12 may refer.
After all of the examinees' tests are graded, statistical and other processing may be provided for various reasons. For instance, to assess an examinee's score, it is necessary to compare his or her score to those of other examinees taking the same test. Another important reason to evaluate the test results for statistical purposes is to create and update an information bank containing the performance statistics of each item used or created for previous tests. This information may then be used for the creation and assembly of future tests.
A goal of standardized testing is to efficiently construct a test for the purpose of measuring a skill, ability, etc. Therefore, each test is constructed to conform to a test specification which defines the rules and/or constraints for selecting the items. In constructing a test, test developers select items from inventory of items so that the combination of selected items satisfy the test specification.
A test is typically divided into sections of questions. The test specification generally defines the number of items to be presented in the test, the number of test sections, the number of questions in each section, the time for taking the test, and the allotted time for responding to all the items in each test section. The test specification also specifies criteria for item selection. These are based on at least four item characteristics which include: (1) item content, e.g., mathematical questions relating to arithmetic, algebra, or geometry; (2) cross-information among items, e.g., more than one item testing the same point; (3) number of items/set, i.e., identification of a subset of items of a larger set; and (4) statistical properties of items derived from pretesting, e.g. difficulty of the selected items.
In recent years, the methods for creating, delivering, administering, and scoring tests have been determined to be inadequate. Due to the number of examinees taking standardized tests, there is increasing demand for developing new and more diverse tests and an increasing need to provide more flexibility in scheduling tests without sacrificing administration costs and security. One solution to these demands would be to automate the entire testing process. Only a few attempts have been made, however, to automate only portions of the testing process. Furthermore, these attempts are limited in their ability to select items from a given inventory of items.
For example, a prior art computerized testing system, "TD/DC," is disclosed in U.S. Pat. No. 5,565,316, entitled "System and Method for Computer Based Testing" and assigned to Educational Testing Service, Princeton, N.J. The TD/DC system, however, has drawbacks in that the assembly stage is not automated. During the test assembly or packaging stage of the TD/DC system, much of the work has to be done manually through the use of paper copies and work folders. In particular, an assembler pulls the appropriate number of each type of item using paper copies and sequences the items in a work folder. Editors work with marked-up paper copies and assemblers have to check copies sent to the printer with the paper originals.
Basically, the prior art system has numerous poorly-integrated steps. Additionally, those steps which are automated involve several software packages running on incompatible platforms such as Unix, OS2, Macintosh, and DOS. Because reviews and changes occurred throughout the process and reviews largely occurred from paper copies, steps were necessitated to make changes to electronic files and to keep the files in all the systems reconciled.
Because the entire test assembly process is not automated, the TD/DC system involves many steps and "hand-offs" of the item/work folder. With so many hand-offs, much time is wasted waiting for a user to pass the work folder off to the next user. For example, on the math section of the SAT test, only 22% of the elapsed time to create an item is actually spent working on that test. Moreover, on the GRE Chemistry test, 30% of the total working time is spent making transcriptions between paper and computer. An automated test assembly system is desired which will streamline the test assembly process by making the process more efficient and reducing the required number of steps and hand-offs.
Another drawback of the prior art systems is that item selection is not efficient. With the prior art system, a user could get 500 test items from a search for test items with particular characteristics. Manually assembling a test with about 50 test items from inventory of 500 is a formidable task. Thus, it is desired to make the item selection process more efficient and automated, allowing users to manipulate pools of items on-line, and thus achieve a finer level of granularity in narrowing down the number of test items in a given inventory before assembling a test.
Another drawback of the prior art test assembly system is that the item selection process had major handicaps in that it had limited information on each test item in its inventory of available items. It is thus desired to provide an inventory of available items with enough data on each item to allow for a more thorough, efficient and finer level of searching and selection.
The main object of the present invention is to address the above-noted problems with the prior art test assembly systems by providing a more efficient test assembly system. The prior art system, i.e., the TD/DC system, lacks efficiency in that much of it is still paper based, it involves numerous steps and hand-offs, and it lacks automatic or automated steps. Thus, the primary object is to improve all aspects of the test assembly system so as to yield an automated and efficient computer-based test assembly system with several automatic features.